Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization

[ES] Hoy en día, la sociedad tiene acceso y posibilidad de contribuir a grandes cantidades de contenidos presentes en Internet, como redes sociales, periódicos online, foros, blogs o plataformas de contenido multimedia. Todo este tipo de medios han tenido, durante los últimos años, un impacto abrumador en el día a día de individuos y organizaciones, siendo actualmente medios predominantes para compartir, debatir y analizar contenidos online. Por este motivo, resulta de interés trabajar sobre este tipo de plataformas, desde diferentes puntos de vista, bajo el paraguas del Procesamiento del Lenguaje Natural. En esta tesis nos centramos en dos áreas amplias dentro de este campo, aplicadas al análisis de contenido en línea: análisis de texto en redes sociales y resumen automático. En paralelo, las redes neuronales también son un tema central de esta tesis, donde toda la experimentación se ha realizado utilizando enfoques de aprendizaje profundo, principalmente basados en mecanismos de atención. Además, trabajamos mayoritariamente con el idioma español, por ser un idioma poco explorado y de gran interés para los proyectos de investigación en los que participamos. Por un lado, para el análisis de texto en redes sociales, nos enfocamos en tareas de análisis afectivo, incluyendo análisis de sentimientos y detección de emociones, junto con el análisis de la ironía. En este sentido, se presenta un enfoque basado en Transformer Encoders, que consiste en contextualizar \textit{word embeddings} pre-entrenados con tweets en español, para abordar tareas de análisis de sentimiento y detección de ironía. También proponemos el uso de métricas de evaluación como funciones de pérdida, con el fin de entrenar redes neuronales, para reducir el impacto del desequilibrio de clases en tareas \textit{multi-class} y \textit{multi-label} de detección de emociones. Adicionalmente, se presenta una especialización de BERT tanto para el idioma español como para el dominio de Twitter, que tiene en cuenta la coherencia entre tweets en conversaciones de Twitter. El desempeño de todos estos enfoques ha sido probado con diferentes corpus, a partir de varios \textit{benchmarks} de referencia, mostrando resultados muy competitivos en todas las tareas abordadas. Por otro lado, nos centramos en el resumen extractivo de artículos periodísticos y de programas televisivos de debate. Con respecto al resumen de artículos, se presenta un marco teórico para el resumen extractivo, basado en redes jerárquicas siamesas con mecanismos de atención. También presentamos dos instancias de este marco: \textit{Siamese Hierarchical Attention Networks} y \textit{Siamese Hierarchical Transformer Encoders}. Estos sistemas han sido evaluados en los corpora CNN/DailyMail y NewsRoom, obteniendo resultados competitivos en comparación con otros enfoques extractivos coetáneos. Con respecto a los programas de debate, se ha propuesto una tarea que consiste en resumir las intervenciones transcritas de los ponentes, sobre un tema determinado, en el programa "La Noche en 24 Horas". Además, se propone un corpus de artículos periodísticos, recogidos de varios periódicos españoles en línea, con el fin de estudiar la transferibilidad de los enfoques propuestos, entre artículos e intervenciones de los participantes en los debates. Este enfoque muestra mejores resultados que otras técnicas extractivas, junto con una transferibilidad de dominio muy prometedora.[CA] Avui en dia, la societat té accés i possibilitat de contribuir a grans quantitats de continguts presents a Internet, com xarxes socials, diaris online, fòrums, blocs o plataformes de contingut multimèdia. Tot aquest tipus de mitjans han tingut, durant els darrers anys, un impacte aclaparador en el dia a dia d'individus i organitzacions, sent actualment mitjans predominants per compartir, debatre i analitzar continguts en línia. Per aquest motiu, resulta d'interès treballar sobre aquest tipus de plataformes, des de diferents punts de vista, sota el paraigua de l'Processament de el Llenguatge Natural. En aquesta tesi ens centrem en dues àrees àmplies dins d'aquest camp, aplicades a l'anàlisi de contingut en línia: anàlisi de text en xarxes socials i resum automàtic. En paral·lel, les xarxes neuronals també són un tema central d'aquesta tesi, on tota l'experimentació s'ha realitzat utilitzant enfocaments d'aprenentatge profund, principalment basats en mecanismes d'atenció. A més, treballem majoritàriament amb l'idioma espanyol, per ser un idioma poc explorat i de gran interès per als projectes de recerca en els que participem. D'una banda, per a l'anàlisi de text en xarxes socials, ens enfoquem en tasques d'anàlisi afectiu, incloent anàlisi de sentiments i detecció d'emocions, juntament amb l'anàlisi de la ironia. En aquest sentit, es presenta una aproximació basada en Transformer Encoders, que consisteix en contextualitzar \textit{word embeddings} pre-entrenats amb tweets en espanyol, per abordar tasques d'anàlisi de sentiment i detecció d'ironia. També proposem l'ús de mètriques d'avaluació com a funcions de pèrdua, per tal d'entrenar xarxes neuronals, per reduir l'impacte de l'desequilibri de classes en tasques \textit{multi-class} i \textit{multi-label} de detecció d'emocions. Addicionalment, es presenta una especialització de BERT tant per l'idioma espanyol com per al domini de Twitter, que té en compte la coherència entre tweets en converses de Twitter. El comportament de tots aquests enfocaments s'ha provat amb diferents corpus, a partir de diversos \textit{benchmarks} de referència, mostrant resultats molt competitius en totes les tasques abordades. D'altra banda, ens centrem en el resum extractiu d'articles periodístics i de programes televisius de debat. Pel que fa a l'resum d'articles, es presenta un marc teòric per al resum extractiu, basat en xarxes jeràrquiques siameses amb mecanismes d'atenció. També presentem dues instàncies d'aquest marc: \textit{Siamese Hierarchical Attention Networks} i \textit{Siamese Hierarchical Transformer Encoders}. Aquests sistemes s'han avaluat en els corpora CNN/DailyMail i Newsroom, obtenint resultats competitius en comparació amb altres enfocaments extractius coetanis. Pel que fa als programes de debat, s'ha proposat una tasca que consisteix a resumir les intervencions transcrites dels ponents, sobre un tema determinat, al programa "La Noche en 24 Horas". A més, es proposa un corpus d'articles periodístics, recollits de diversos diaris espanyols en línia, per tal d'estudiar la transferibilitat dels enfocaments proposats, entre articles i intervencions dels participants en els debats. Aquesta aproximació mostra millors resultats que altres tècniques extractives, juntament amb una transferibilitat de domini molt prometedora.[EN] Nowadays, society has access, and the possibility to contribute, to large amounts of the content present on the internet, such as social networks, online newspapers, forums, blogs, or multimedia content platforms. These platforms have had, during the last years, an overwhelming impact on the daily life of individuals and organizations, becoming the predominant ways for sharing, discussing, and analyzing online content. Therefore, it is very interesting to work with these platforms, from different points of view, under the umbrella of Natural Language Processing. In this thesis, we focus on two broad areas inside this field, applied to analyze online content: text analytics in social media and automatic summarization. Neural networks are also a central topic in this thesis, where all the experimentation has been performed by using deep learning approaches, mainly based on attention mechanisms. Besides, we mostly work with the Spanish language, due to it is an interesting and underexplored language with a great interest in the research projects we participated in. On the one hand, for text analytics in social media, we focused on affective analysis tasks, including sentiment analysis and emotion detection, along with the analysis of the irony. In this regard, an approach based on Transformer Encoders, based on contextualizing pretrained Spanish word embeddings from Twitter, to address sentiment analysis and irony detection tasks, is presented. We also propose the use of evaluation metrics as loss functions, in order to train neural networks for reducing the impact of the class imbalance in multi-class and multi-label emotion detection tasks. Additionally, a specialization of BERT both for the Spanish language and the Twitter domain, that takes into account inter-sentence coherence in Twitter conversation flows, is presented. The performance of all these approaches has been tested with different corpora, from several reference evaluation benchmarks, showing very competitive results in all the tasks addressed. On the other hand, we focused on extractive summarization of news articles and TV talk shows. Regarding the summarization of news articles, a theoretical framework for extractive summarization, based on siamese hierarchical networks with attention mechanisms, is presented. Also, we present two instantiations of this framework: Siamese Hierarchical Attention Networks and Siamese Hierarchical Transformer Encoders. These systems were evaluated on the CNN/DailyMail and the NewsRoom corpora, obtaining competitive results in comparison to other contemporary extractive approaches. Concerning the TV talk shows, we proposed a text summarization task, for summarizing the transcribed interventions of the speakers, about a given topic, in the Spanish TV talk shows of the ``La Noche en 24 Horas" program. In addition, a corpus of news articles, collected from several Spanish online newspapers, is proposed, in order to study the domain transferability of siamese hierarchical approaches, between news articles and interventions of debate participants. This approach shows better results than other extractive techniques, along with a very promising domain transferability.González Barba, JÁ. (2021). Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/172245TESI

Aprendizaje profundo para el procesamiento del lenguaje natural

The work carried out is part of natural language processing area and it is focused specifically on text classification problems like sentiment analysis and topic detection about content published on Twitter, using deep learning models such as convolutional and recurrent neural networks. In order to estimate the parameters of these models in problems where the input is a sequence of linguistic units, a representation that retains the most discriminative information for a given task (contextual information, semantic, syntactic, etc.) is required. For this reason, in the present work, suitable representations for the addressed tasks are proposed and comparisons are made among the different representations, as well as among the different models of neural networks that use these representations. All the tasks addressed have been proposed in workshops of national and international congresses such as SemEval and IberEval. Furthermore, in these tasks we have obtained competitive results, reaching a 1st place in the tasks COSET and Gender (Ibereval 2017@SEPLN), a 4th place in Stance (Ibereval 2017@SEPLN) and a 3rd place in Sentiment Analysis in Twitter (SemEval 2017@ACL).El trabajo realizado se enmarca en el área del procesamiento del lenguaje natural y se centra, concretamente, en problemas de clasificación de texto como sentiment analysis y topic detection sobre contenido publicado en Twitter, haciendo uso de modelos basados en aprendizaje profundo como redes convolucionales y redes recurrentes. Para poder estimar los parámetros de estos modelos, en problemas donde la entrada es una secuencia de unidades lingüísticas, se requiere una representación que retenga la información más discriminativa para una tarea determinada (información contextual, semántica, sintáctica, entre otras). Por este motivo, en el presente trabajo se proponen representaciones adecuadas para las tareas tratadas y se realizan comparaciones entre las diferentes representaciones, así como entre los diferentes modelos de redes neuronales que hacen uso de dichas representaciones. Todas las tareas abordadas han sido propuestas en talleres de congresos nacionales e internacionales como SemEval e IberEval. Además, en dichas tareas se han obtenido resultados competitivos, llegando a alcanzar un 1º puesto en las tareas COSET y Gender (Ibereval 2017@SEPLN), un 4º puesto en Stance (Ibereval 2017@SEPLN) y un 3º puesto en Sentiment Analysis in Twitter (SemEval 2017@ACL).[CA] El treball realitzat s’emmarca en l’`area del processament del llenguatge natural i es centra, concretament, en problemes de classificaci´o de text com sentiment analysis i topic detection sobre contingut publicat en Twitter, fent ´us de models basats en aprenentatge profund com xarxes convolucionals i xarxes recurrents. Per poder estimar els par`ametres d’aquests models, en problemes on l’entrada ´es una seq¨u`encia d’unitats ling¨u´ıstiques, es requereix una representaci´o que retinga la informaci´o m´es discriminativa per a una determinada tasca (informaci´o contextual, sem`antica, sint`actica, entre altres). Per aquest motiu, en el present treball es proposen representacions adequades per las tasques tractades i es realitzen comparacions entre les diferents representacions, aix´ı com entre els diferents models de xarxes neuronals que fan ´us d’aquestes representacions. Totes les tasques abordades han sigut propostes en tallers de congressos nacionals i internacionals com SemEval i IberEval. A m´es, en aquestes tasques s’han obtingut resultats competitius, aconseguint un 1r lloc en les tasques COSET i Gender (Ibereval 2017@SEPLN ), un 4t lloc en Stance (Ibereval 2017@SEPLN ) i un 3r lloc en Sentiment Analysis in Twitter (SemEval 2017@ACL).González Barba, JÁ. (2017). Aprendizaje profundo para el procesamiento del lenguaje natural. http://hdl.handle.net/10251/86279TFG

Descubrimiento automático de conocimiento

[ES] En el presente proyecto se proponen, evalúan y estudian soluciones basadas en representaciones vectoriales continuas y discretas de palabras y frases a algunos de los problemas más significativos del descubrimiento automático de conocimiento aplicado al lenguaje natural y en general a lenguajes formales. Entre estos problemas destacan la detección de temática, identificación de idioma, análisis de sentimiento y detección de malware. Además, debido a la complejidad que supone el aprendizaje y la utilización de dichas representaciones vectoriales, se ha desarrollado un sistema que facilita las tareas de evaluación, preprocesamiento, extracción de características y visualización de resultados; generalizando los aspectos comunes a todos los problemas abordados. Destacamos, también, los buenos resultados obtenidos mediante el empleo de las representaciones mencionadas sobre el problema de detección de temática, que constituye el principal problema del proyecto, superando los mejores resultados conocidos, haciendo uso del mismo corpus, que han sido cosechados por investigadores de la Universidad Politécnica de Madrid.[EN] In this project, solutions based on continuous and discrete vector representations of word and sentences are proposed, evaluated and studied by using them in some of the most significant problems in automatic knowledge discovery applied to natural language and generally to formal languages. Among these problems, we highlight topic detection, language identification, sentiment analysis and malware detection. Furthermore, due to the complexity of learning and use of vector representations, a system that facilitates evaluation tasks, preprocessing, feature extraction and results display has been developed; generalizing this way the common aspects to all the addressed problems. We highlight too the good results obtained by means of using these representations on topic detection, which is the main problem of the project, surpassing the best known results that have been reached by Polytechnic University of Madrid researchers which uses the same corpus.[CA] En el present projecte es proposen, avaluen i estudien solucions basades en representacions vectorials contínues i discretes de paraules i frases a alguns dels problemes més significatius del descobriment automàtic de coneixement aplicat al llenguatge natural i en general a llenguatges formals. Entre aquests problemes destaquem la detecció de temàtica, identificació d’idiomes, anàlisi de sentiment i detecció de malware. A més, a causa de la complexitat que suposa l’aprenentatge i la utilització d’aquestes representacions vectorials, s’ha implementat un sistema que facilita les tasques d’avaluació, preprocessament, extracció de característiques i visualització de resultats; generalitzant els aspectes en comú a tots els problemes abordats. Destaquem, també, els bons resultats obtinguts mitjançant l’ús d’aquestes representacions en el problema de detecció de temàtica, que constitueix el principal problema del projecte i se superen els millors resultats coneguts, fent ús del mateix corpus, que han sigut aconseguits per investigadors de la Universitat Politècnica de Madrid.González Barba, JÁ. (2016). Descubrimiento automático de conocimiento. http://hdl.handle.net/10251/69227.TFG

Transformer based contextualization of pre-trained word embeddings for irony detection in Twitter

[EN] Human communication using natural language, specially in social media, is influenced by the use of figurative language like irony. Recently, several workshops are intended to explore the task of irony detection in Twitter by using computational approaches. This paper describes a model for irony detection based on the contextualization of pre-trained Twitter word embeddings by means of the Transformer architecture. This approach is based on the same powerful architecture as BERT but, differently to it, our approach allows us to use in-domain embeddings. We performed an extensive evaluation on two corpora, one for the English language and another for the Spanish language. Our system was the first ranked system in the Spanish corpus and, to our knowledge, it has achieved the second-best result on the English corpus. These results support the correctness and adequacy of our proposal. We also studied and interpreted how the multi-head self-attention mechanisms are specialized on detecting irony by means of considering the polarity and relevance of individual words and even the relationships among words. This analysis is a first step towards understanding how the multi-head self-attention mechanisms of the Transformer architecture address the irony detection problem.This work has been partially supported by the Spanish Ministerio de Ciencia, Innovacion y Universidades and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and the GiSPRO project (PROMETEU/2018/176). Work of Jose-Angel Gonzalez is financed by Universitat Politecnica de Valencia under grant PAID-01-17.González-Barba, JÁ.; Hurtado Oliver, LF.; Pla Santamaría, F. (2020). Transformer based contextualization of pre-trained word embeddings for irony detection in Twitter. Information Processing & Management. 57(4):1-15. https://doi.org/10.1016/j.ipm.2020.102262S115574Farías, D. I. H., Patti, V., & Rosso, P. (2016). Irony Detection in Twitter. ACM Transactions on Internet Technology, 16(3), 1-24. doi:10.1145/2930663Greene, R., Cushman, S., Cavanagh, C., Ramazani, J., & Rouzer, P. (Eds.). (2012). The Princeton Encyclopedia of Poetry and Poetics. doi:10.1515/9781400841424Van Hee, C., Lefever, E., & Hoste, V. (2018). We Usually Don’t Like Going to the Dentist: Using Common Sense to Detect Irony on Twitter. Computational Linguistics, 44(4), 793-832. doi:10.1162/coli_a_00337Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.1735Joshi, A., Bhattacharyya, P., & Carman, M. J. (2017). Automatic Sarcasm Detection. ACM Computing Surveys, 50(5), 1-22. doi:10.1145/3124420Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations.Mohammad, S. M., & Turney, P. D. (2012). CROWDSOURCING A WORD-EMOTION ASSOCIATION LEXICON. Computational Intelligence, 29(3), 436-465. doi:10.1111/j.1467-8640.2012.00460.xMuecke, D. C. (1978). Irony markers. Poetics, 7(4), 363-375. doi:10.1016/0304-422x(78)90011-6Potamias, R. A., Siolas, G., & Stafylopatis, A. (2019). A transformer-based approach to irony and sarcasm detection. arXiv:1911.10401.Rosso, P., Rangel, F., Farías, I. H., Cagnina, L., Zaghouani, W., & Charfi, A. (2018). A survey on author profiling, deception, and irony detection for the Arabic language. Language and Linguistics Compass, 12(4), e12275. doi:10.1111/lnc3.12275Sulis, E., Irazú Hernández Farías, D., Rosso, P., Patti, V., & Ruffo, G. (2016). Figurative messages and affect in Twitter: Differences between #irony, #sarcasm and #not. Knowledge-Based Systems, 108, 132-143. doi:10.1016/j.knosys.2016.05.035Wilson, D., & Sperber, D. (1992). On verbal irony. Lingua, 87(1-2), 53-76. doi:10.1016/0024-3841(92)90025-eYus, F. (2016). Propositional attitude, affective attitude and irony comprehension. Pragmatics & Cognition, 23(1), 92-116. doi:10.1075/pc.23.1.05yusZhang, S., Zhang, X., Chan, J., & Rosso, P. (2019). Irony detection via sentiment-based transfer learning. Information Processing & Management, 56(5), 1633-1644. doi:10.1016/j.ipm.2019.04.00

Choosing the right loss function for multi-label Emotion Classification

[EN] Natural Language Processing problems has recently been benefited for the advances in Deep Learning. Many of these problems can be addressed as a multi-label classification problem. Usually, the metrics used to evaluate classification models are different from the loss functions used in the learning process. In this paper, we present a strategy to incorporate evaluation metrics in the learning process in order to increase the performance of the classifier according to the measure we are interested to favor. Concretely, we propose soft versions of the Accuracy, micro-F-1, and macro-F-1 measures that can be used as loss functions in the back-propagation algorithm. In order to experimentally validate our approach, we tested our system in an Emotion Classification task proposed at the International Workshop on Semantic Evaluation, SemEval-2018. Using a Convolutional Neural Network trained with the proposed loss functions we obtained significant improvements both for the English and the Spanish corpora.This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and the GiSPRO project (PROMETEU/2018/176). Work of Jose-Angel Gonzalez is also financed by Universitat Politecnica de Valencia under grant PAID-01-17.Hurtado Oliver, LF.; González-Barba, JÁ.; Pla Santamaría, F. (2019). Choosing the right loss function for multi-label Emotion Classification. Journal of Intelligent & Fuzzy Systems. 36(5):4697-4708. https://doi.org/10.3233/JIFS-179019S46974708365Baccianella S. , Esuli A. and Sebastiani F. , Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, In in Proc of LREC, 2010.Bilmes J. , Asanovic K. , Chin C.-W. and Demmel J. , Using phipac to speed error back-propagation learning, In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, 1997, pp. 4153–4156.Cruz, F. L., Troyano, J. A., Pontes, B., & Ortega, F. J. (2014). Building layered, multilingual sentiment lexicons at synset and lemma levels. Expert Systems with Applications, 41(13), 5984-5994. doi:10.1016/j.eswa.2014.04.005Dembczynski K. , Jachnik A. , Kotlowski W. , Waegeman W. and Huellermeier E. , Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization, In DasguptaS. and McAllester D., editors, Proceedings of the 30th International Conference on Machine Learning volume 28 of Proceedings of Machine Learning Research, Atlanta, Georgia, USA, PMLR, 2013, pp. 1130–1138.Goodfellow I. , Bengio Y. and Courville A. , Deep Learning, MIT Press, http://www.deeplearningbook.org (2016).Hu M. and Liu B. , Mining and summarizing customer reviews, In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, New York, NY, USA, ACM, 2004, pp. 168–177.Ioffe S. and Szegedy C. , Batch normalization: Accelerating deep network training by reducing internal covariate shift, CoRR, abs/1502.03167 (2015).Janocha K. and Czarnecki W.M. , On loss functions for deep neural networks in classification, CoRR, abs/1702.05659 (2017).Krieger M. and Ahn D. , Tweetmotif: Exploratory search and topic summarization for twitter, In Proc of AAAI Conference on Weblogs and Social, 2010.Liu B. , Sentiment Analysis and Opinion Mining, A Comprehensive Introduction and Survey. Morgan & Claypool Publishers, 2012.Mikolov T. , Sutskever I. , Chen K. , Corrado G. and Dean J. , Distributed representations of words and phrases and their compositionality, CoRR, abs/1310.4546 (2013a).Mikolov T. , Chen K. , Corrado G. and Dean J. , Efficient estimation of word representations in vector space, CoRR, abs/1301.3781, 2013b.Mohammad S. , #emotional tweets, In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the Main Conference and the Shared Task and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, Canada. Association for Computational Linguistics, 2012, pp. 246–255.Mohammad S. , Kiritchenko S. , Sobhani P. , Zhu X. and Cherry C. , Semeval-task 6: Detecting stance in tweets, In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 31–41.Mohammad S.M. and Bravo-Marquez F. , WASSA-shared task on emotion intensity, CoRR, abs/1708.03700, 2017.Mohammad, S. M., & Turney, P. D. (2012). CROWDSOURCING A WORD-EMOTION ASSOCIATION LEXICON. Computational Intelligence, 29(3), 436-465. doi:10.1111/j.1467-8640.2012.00460.xMohammad, S. M., Sobhani, P., & Kiritchenko, S. (2017). Stance and Sentiment in Tweets. ACM Transactions on Internet Technology, 17(3), 1-23. doi:10.1145/3003433Mohammad S.M. , Bravo-Marquez F. , Salameh M. and Kiritchenko S. , Semeval-2018 Task 1: Affect in tweets, In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, 2018.Molina-González, M. D., Martínez-Cámara, E., Martín-Valdivia, M.-T., & Perea-Ortega, J. M. (2013). Semantic orientation for polarity classification in Spanish reviews. Expert Systems with Applications, 40(18), 7250-7257. doi:10.1016/j.eswa.2013.06.076Nair V. and Hinton G.E. , Rectified linear units improve restricted boltzmann machines, In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, USA, 2010, pp. 807–814. Omnipress.NielsenF.Å., AFINN, 2011.Pastor-Pellicer J. , Zamora-Martínez F. , España Boquera S. and Castro Bleda M.J. , F-Measure as the Error Function to Train Neural Networks, In IWANN Proceedings, 2013.Pennebaker J. , Chung C. , Ireland M. , Gonzales A. and Booth R. , The development and psychological properties of liwc2007, 2014.Pla, F., & Hurtado, L.-F. (2016). Language identification of multilingual posts from Twitter: a case study. Knowledge and Information Systems, 51(3), 965-989. doi:10.1007/s10115-016-0997-xRosenthal S. , Farra N. and Nakov P. , SemEval-2017 task 4: Sentiment analysis in Twitter, In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, Vancouver, Canada, Association for Computational Linguistics, 2017.Saralegi X. and San I. , Vicente, Elhuyar at tass 2013, In XXIX Congreso de la Sociedad Espaola de Procesamiento de Lenguaje Natural, Workshop on Sentiment Analysis at SEPLN (TASS2013), 2013, pp. 143–150.Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1-47. doi:10.1145/505282.505283Taulé M. , Martí M. , Rangel F. , Rosso P. , Bosco C. and Patti V. , Overview of the task of Stance and Gender Detection in Tweets on Catalan Independence at IBEREVAL 2017, In Notebook Papers of 2nd SEPLN Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL), Murcia (Spain). CEUR Workshop Proceedings. CEUR-WS.org, 2017, 2017.Wiebe J. , Wilson T. and Cardie C. , Annotating expressions of opinions and emotions in language, Language Resources and Evaluation 1(2) (2005).Wilson T. , Wiebe J. and Hoffmann P. , Recognizing contextual polarity in phrase-level sentiment analysis, In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, Stroudsburg, PA, USA, 2005, pp. 347–354. Association for Computational Linguistics.Zhang Y. and Wallace B. , A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 253–263. Asian Federation of Natural Language Processing

Self-attention for Twitter sentiment analysis in Spanish

[EN] This paper describes our proposal for Sentiment Analysis in Twitter for the Spanish language. The main characteristics of the system are the use of word embedding specifically trained from tweets in Spanish and the use of self-attention mechanisms that allow to consider sequences without using convolutional nor recurrent layers. These self-attention mechanisms are based on the encoders of the Transformer model. The results obtained on the Task 1 of the TASS 2019 workshop, for all the Spanish variants proposed, support the correctness and adequacy of our proposal.This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and the GiSPRO project (PROMETEU/2018/176). Work of Jose-Angel Gonzalez is financed by Universitat Politecnica de Valencia under grant PAID-01-17.González-Barba, JÁ.; Hurtado Oliver, LF.; Pla Santamaría, F. (2020). Self-attention for Twitter sentiment analysis in Spanish. Journal of Intelligent & Fuzzy Systems. 39(2):2165-2175. https://doi.org/10.3233/JIFS-179881S21652175392Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.173

ELiRF-UPV at TASS 2020: TWilBERT for Sentiment Analysis and Emotion Detection in Spanish Tweets

[EN] This paper describes the participation of the ELiRF research group of the Universitat Politècnica de València in the TASS 2020 Workshop, framed within the XXXVI edition of the International Conference of the Spanish Society for the Processing of Natural Language (SEPLN). We present the approach used for the Monolingual Sentiment Analysis and Emotion Detection tasks of the workshop, as well as the results obtained. Our participation has focused mainly on employing an adaptation of BERT for text classification on the Twitter domain and the Spanish language. This system, that we have called TWilBERT, shown systematic improvements of the state of the art in almost all the tasks framed in the SEPLN conference of previous years, and also obtains the most competitive performance in the tasks addressed in this work.This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and by the GiSPRO project (PROMETEU/2018/176). Work of José-Ángel González is financed by Universitat Politècnica de València under grant PAID-01-17.González-Barba, JÁ.; Arias-Moncho, J.; Hurtado Oliver, LF.; Pla Santamaría, F. (2020). ELiRF-UPV at TASS 2020: TWilBERT for Sentiment Analysis and Emotion Detection in Spanish Tweets. CEUR. 179-186. http://hdl.handle.net/10251/17855817918

NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

[EN] Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called content reordering, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.This work was partially supported by the Spanish Ministerio de Ciencia, Innovacion y Universidades and FEDER founds under the project AMIC (TIN2017-85854-C4-2-R), and by the Agencia Valenciana de la Innovacio (AVI) of the Generalitat Valenciana under the GUAITA (INNVA1/2020/61) project.Ahuir-Esteve, V.; Hurtado Oliver, LF.; González-Barba, JÁ.; Segarra Soriano, E. (2021). NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish. Applied Sciences. 11(21):1-16. https://doi.org/10.3390/app11219872S116112

Applying Siamese Hierarchical Attention Neural Networks for multi-document summarization

[EN] In this paper, we present an approach to multi-document summarization based on Siamese Hierarchical Attention Neural Networks. The attention mechanism of Hierarchical Attention Networks, provides a score to each sentence in function of its relevance in the classification process. For the summarization process, only the scores of sentences are used to rank them and select the most salient sentences. In this work we explore the adaptability of this model to the problem of multi-document summarization (typically very long documents where the straightforward application of neural networks tends to fail). The experiments were carried out using the CNN/DailyMail as training corpus, and the DUC-2007 as test corpus. Despite the difference between training set (CNN/DailyMail) and test set (DUC-2007) characteristics, the results show the adequacy of this approach to multi-document summarization.This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R). Work of Jose-Angel Gonzalez is also financed by Universitat Politecnica de Valencia under grant PAID-01-17.González-Barba, JÁ.; Julien Delonca; Sanchís Arnal, E.; García-Granada, F.; Segarra Soriano, E. (2019). Applying Siamese Hierarchical Attention Neural Networks for multi-document summarization. PROCESAMIENTO DEL LENGUAJE NATURAL. (63):111-118. https://doi.org/10.26342/2019-63-12S1111186

Extractive summarization using siamese hierarchical transformer encoders

[EN] In this paper, we present an extractive approach to document summarization, the Siamese Hierarchical Transformer Encoders system, that is based on the use of siamese neural networks and the transformer encoders which are extended in a hierarchical way. The system, trained for binary classification, is able to assign attention scores to each sentence in the document. These scores are used to select the most relevant sentences to build the summary. The main novelty of our proposal is the use of self-attention mechanisms at sentence level for document summarization, instead of using only attentions at word level. The experimentation carried out using the CNN/DailyMail summarization corpus shows promising results in-line with the state-of-the-art.This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R). Work of Jose Angel Gonzalez is also financed by Universitat Politecnica de Valencia under grant PAID-01-17.González-Barba, JÁ.; Segarra Soriano, E.; García-Granada, F.; Sanchís Arnal, E.; Hurtado Oliver, LF. (2020). Extractive summarization using siamese hierarchical transformer encoders. Journal of Intelligent & Fuzzy Systems. 39(2):2409-2419. https://doi.org/10.3233/JIFS-179901S24092419392Begum N. , Fattah M. and Ren F. , Automatic text summarization using support vector machine, 5 (2009), 1987–1996.González, J.-Á., Segarra, E., García-Granada, F., Sanchis, E., & Hurtado, L.-F. (2019). Siamese hierarchical attention networks for extractive summarization. Journal of Intelligent & Fuzzy Systems, 36(5), 4599-4607. doi:10.3233/jifs-179011Lloret, E., & Palomar, M. (2011). Text summarisation in progress: a literature review. Artificial Intelligence Review, 37(1), 1-41. doi:10.1007/s10462-011-9216-zLouis, A., & Nenkova, A. (2013). Automatically Assessing Machine Summary Content Without a Gold Standard. Computational Linguistics, 39(2), 267-300. doi:10.1162/coli_a_00123Tur G. and De Mori R. , Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons, 2011